feat(bigquery-jdbc): add EnableProjectDiscovery connection property for metadata methods#13344
feat(bigquery-jdbc): add EnableProjectDiscovery connection property for metadata methods#13344keshavdandeva wants to merge 8 commits into
EnableProjectDiscovery connection property for metadata methods#13344Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces an EnableProjectDiscovery configuration property to automatically discover and list all accessible Google Cloud projects as catalogs. To support this, schema fetching in BigQueryDatabaseMetaData has been refactored to run concurrently using an executor service. Feedback on these changes highlights two main areas for improvement: first, replacing the fragile use of reflection to access the low-level BigQuery client with the standard public BigQuery.listProjects() API; second, ensuring that outstanding asynchronous tasks are properly cancelled if the schema fetching loop is interrupted to prevent resource leaks.
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces an automatic project discovery feature to the BigQuery JDBC driver, allowing users to discover all accessible Google Cloud projects via a new EnableProjectDiscovery connection property. It also updates BigQueryDatabaseMetaData to parallelize schema fetching across multiple projects. The review feedback highlights several areas for improvement: reusing a shared ExecutorService instead of creating a new one on every getSchemas() call, avoiding caching empty results on transient exceptions in getDiscoveredProjects(), using GsonFactory.getDefaultInstance() for better resource reuse, and preserving the stack trace when logging execution exceptions.
| } | ||
|
|
||
| @Override | ||
| public ResultSet getSchemas(String catalog, String schemaPattern) { |
There was a problem hiding this comment.
I'd suggest some refactoring for this method, few ideas to simplify:
- Move code to fetch list of schemas in a specific catalog to a separate function;
- Keep this method simple - single catalog only, essentially calls into helper & transforms to jsonResultSet. No need for background threads since it is single catalog
- Refactor
getSchemas()to be the one that fans out multiple requests & assembles data.
Probably breaking up CLs in 2 will be easier:
- Add proper threadPool to connection-layer (and remove static threadPool in statement, it is not used)
- update getSchemas
Also I'd suggest to reuse more code. There are 4 metadata methods that can generate large # of rest calls:
- getCatalogs()
- getSchemas()
- getTables()
- getColumns()
- getProcedures()
In a way, they build results on top of each other. Right now we duplicate a lot, e.g. listDatasets() is called in 7 different places.
Also please don't tackle all of them in a single CL, lets do it incrementally :)
(Otherwise it is pain to review)
There was a problem hiding this comment.
Yeah, makes sense. I have created 3 bugs:
- b/520400589 - Refactor Metadata Thread Pool Management to Reuse Connection-Scoped Executor
- b/520407325 - Refactor getSchemas for Catalog-Based Routing (Synchronous & Async Fan-out)
- b/520406763 - Deduplicate metadata API calls
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces an automatic project discovery feature to the BigQuery JDBC driver, allowing users to discover all accessible Google Cloud projects via the EnableProjectDiscovery connection property. The implementation includes caching of discovered projects and error handling for transient and non-transient BigQuery exceptions. The review feedback correctly identifies compilation errors in BigQueryConnection.java where invalid LOG.warning(e, ...) calls are used instead of LOG.log(Level.WARNING, ..., e).
3 participants
b/499078725
This PR implements the
EnableProjectDiscoveryconnection property into the BigQuery JDBC driver. Enabling this property (defaultfalse) allows JDBC database metadata methods (likegetCatalogs()andgetSchemas()) to discover and query datasets across all Google Cloud projects accessible to the client credentials, rather than being confined to the single defaultProjectIdspecified in the connection URL.Changes
EnableProjectDiscoveryinBigQueryJdbcUrlUtilityand configured it inDataSource.BigQueryConnection.getDiscoveredProjects()to utilize theBigQuery.listProjects()core SDK methoddiscoveredProjectsCache.400,401,403) cache an empty list to avoid redundant, failing calls to the BigQuery API.500) are not cached, allowing subsequent metadata scans to trigger retries.BigQueryDatabaseMetaData.getAccessibleCatalogNames()so that catalog and schema metadata fetches automatically query across all accessible projects whenEnableProjectDiscoveryis enabled.